Image search apparatus and image search method

ABSTRACT

An image search apparatus has: a dimension reducing unit for reducing dimensions of multidimensional feature data to be produced as approximate data; an approximate-data storing unit for storing the produced approximate data with corresponding to the multidimensional feature data before the dimension reduction; a search request receiving unit for receiving an identifier which identifies multidimensional feature data of a person to be searched, at least as a search key; an approximate-space searching unit for calculating distances between approximate data corresponding to the received search key, and plural approximate data stored in the approximate-data storing unit, and for arranging the distances in order of similarity; and a real-space final ranking unit for again performing a distance calculation with using the multidimensional feature data before the dimension reduction, on a group of results in which the similarity is high, and for determining final rankings.

This application is a continuation of U.S. patent application Ser. No.12/064,928 filed Feb. 26, 2008 which is incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present invention relates to an image search apparatus and imagesearch method which can sort out a desired image at high speed from animage group of large capacity with using multidimensional feature data(a face, a color, and the like) extracted from an image.

BACKGROUND ART

Recently, with the increase of crimes typified by picking, robbery,arson, and the like, the popularity of a video monitoring system inwhich cameras, sensors, storage devices, and the like are disposed toprevent crimes is largely expanding. In accordance with adoption of IPto a monitoring camera and the increasing of the capacity of a storagedevice, the number of systems which perform wide-area monitoring ofseveral hundred devices and long-time recording is increased. Underthese circumstances, in order to reduce the task of an observer, it hasbeen requested to develop a technique for efficiently sorting out aspecific person such as a shoplifter, a lost child, or a person who haslost an article.

As a conventional technique for sorting out a specific person at highspeed, there is a technique in which a group of multidimensional featuredata (a face, a color, and the like) extracted from an image ispreviously clustered in ascending order of distance to be formed as atree structure, and, in a searching process, only a subtree which isclosest to the person to be searched is subjected to the searchingprocess. Patent Reference 1 discloses a method in which projection isperformed in a model space that is prepared by a statistical technique,and multidimensional feature data that are highly accurate, and thathave a reduced number of dimensions are produced, thereby performing ahigh-speed search.

-   Patent Reference 1: JP-A-2002-183205

DISCLOSURE OF THE INVENTION Problems that the Invention is to Solve

In the conventional tree-structure method, when the number ofmultidimensional feature data is large, however, the number of theadjacent spaces is exponentially increased, and it requires an enormousamount of time to cluster (register) the multidimensional feature data.In the searching process, furthermore, the neighbor search includingadjacent spaces is performed, and therefore an enormous amount of timeis required in the same manner as the registration.

In the technique disclosed in Patent Reference 1, there is a limit tothe dimension reduction due to projection. As a countermeasure againstthis, also a method in which a similarity calculation is performed withusing only high-importance portions such as the eyes/nose/mouth isdescribed. However, portions where personal features appear arevariously different. There is a limit to reduce the dimensions whilemaintaining the accuracy.

The invention has been conducted in view of the above-discussedconventional circumstances. It is an object of the invention to providean image search apparatus and image search method in which, even when alarge amount of multidimensional feature data such as the face/colorexists in high dimensions, an image desired by the user can be searchedefficiently and adequately.

Means for Solving the Problems

The image search apparatus of the invention comprises: dimensionreducing means for reducing dimensions of multidimensional feature dataextracted from an image, to produce approximate data; approximate-datastoring means for storing the approximate data produced by the dimensionreducing means with corresponding to the multidimensional feature databefore the dimension reduction; search request receiving means forreceiving at least an identifier which identifies multidimensionalfeature data of a person to be searched, as a search key;approximate-space searching means for calculating distances betweenapproximate data corresponding to the search key received by the searchrequest receiving means, and plural approximate data stored in theapproximate-data storing means, based on search conditions, and forarranging the distances in order of similarity; and real-space finalranking means for again performing a distance calculation with using themultidimensional feature data before the dimension reduction, on a groupof results in which the similarity obtained by the approximate-spacesearching means is high, for determining final rankings, and foroutputting the final rankings as a search result.

According to the configuration, after search results are refined to someextent in an approximate space where the number of dimensions issuppressed, a final refinement can be performed in a real space, andtherefore, even when the number of dimensions is increased, an imagedesired by the user can be efficiently searched. Since the searchresults are output in order of similarity, the searching process can beperformed more efficiently.

The image search apparatus of the invention is characterized in that thedimension reducing means rearranges elements constituting themultidimensional feature data in descending order of an absolute value,and produces “element numbers, values” of a top N pieces (N: naturalnumber) thereof as the approximate data. The image search apparatus ofthe invention is characterized in that, from a result of a wavelettransform of input multidimensional feature data, the dimension reducingmeans produces values of top N pieces of high-frequency components orlow-frequency components, as the approximate data. The image searchapparatus of the invention is characterized in that multidimensionalfeature data other than the top N pieces of data obtained by thedimension reducing means are produced as (typical values, sign bits),and managed as the approximate data.

According to the configuration, the dimensions can be reduced so that aportion which highly expresses a feature of a person strongly exerts aninfluence. Therefore, the omission of search in a process of refiningthe search result in an approximate space can be reduced.

The image search apparatus of the invention is characterized in that thereal-space final ranking means performs a final ranking process on top Mpieces (M: natural number) in which the similarity obtained by theapproximate-space searching means is high, and outputs top K pieces(K<M) as the search result.

According to the configuration, the omission of search in a process ofrefining the search result in an approximate space can be suppressed.

The image search apparatus of the invention is characterized in that thereal-space final ranking means performs a distance recalculation withusing the multidimensional feature data before the dimension reduction,in descending order of elements in which the similarity obtained by theapproximate-space searching means is high (i.e., in ascending order ofthe approximate distance), completes the process at a timing when top Kpieces of actual distances which are obtained by the recalculation, andin which the similarity is high are smaller than an approximate distanceof all data that are not subjected to the distance recalculation, andoutputs the top K pieces of distances as the search result.

According to the configuration, it is possible to obtain the same resultas the case where the distance calculation is performed with using alldimensions, and therefore the omission of search in a process ofrefining the search result in an approximate space can be made zero.

The image search apparatus of the invention is characterized in that thereal-space final ranking means outputs “factor of distortion due todimension reduction” indicating a degree at which a result obtained bythe approximate-space searching means is changed by the final rankingprocess, as a result.

According to the configuration, it is possible to know the degree atwhich an omission of search occurs in the process of refining the searchresult in an approximate space, and hence it is possible to re-search animage efficiently.

The image search apparatus of the invention is characterized in that theapparatus further comprises re-search condition designating means fordesignating “number of dimensions to be used” and “piece number ofresults to be refined” as search conditions to be used by theapproximate-space searching means.

According to the configuration, even when an omission of search occursin the process of refining the search result in an approximate space,the user can easily perform the refining process due to re-searching.When it is configured so that re-search condition designating meansrefers to “factor of distortion due to dimension reduction” output bythe real-space final ranking means to automatically re-designate thesearch conditions, an image can be searched more efficiently.

The image search apparatus of the invention is characterized in that theapparatus further comprises correctness/incorrectness designating meansfor designating correctness or incorrectness of the search result outputby the real-space final ranking means. The image search apparatus of theinvention is characterized in that the approximate-space searching meanssets element numbers of approximate data which are designated as correctby the correctness/incorrectness designating means, as “correct elementnumber group”, sets element numbers of approximate data which aredesignated as incorrect, as “incorrect element number group”, and, inthe distance calculating process using approximate data, increasesweightings of element numbers included in “correct element numbergroup”, and decreases weightings of element numbers included in“incorrect element number group”.

According to the configuration, even when many omissions of search occurin the process of refining the search result in an approximate space,the user can easily perform the refining operation due to re-searching.

The image search method of the invention includes: approximate-spacesearching step of, with using approximate data produced by reducingdimensions of multidimensional feature data extracted from an image,arranging image groups stored in image storing means in order ofsimilarity with an image to be searched; and real-space final rankingstep of performing a distance re-calculation with using themultidimensional feature data before the dimension reduction, on a groupof results in which the similarity obtained in the approximate-spacesearching step is high, and of determining final rankings.

Effects of the Invention

According to the invention, after search results are refined to someextent in an approximate space where the number of dimensions issuppressed, a final refinement can be performed in a real space. Evenwhen the number of dimensions is increased, therefore, an image desiredby the user can be efficiently searched.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image search apparatus of a firstembodiment of the invention.

FIG. 2 is a flow chart relating to a data registering operation in theimage search apparatus of the first embodiment of the invention.

FIG. 3 is a flow chart relating to a searching operation in the imagesearch apparatus of the first embodiment of the invention.

FIG. 4 is a flow chart relating to a searching operation of eliminatingomission of search in the image search apparatus of the first embodimentof the invention.

FIG. 5 is a diagram relating to a difference between searching processesof two kinds in the image search apparatus of the first embodiment ofthe invention.

FIG. 6 is a flow chart relating to a re-searching operation in an imagesearch apparatus of a second embodiment of the invention.

FIG. 7 is a diagram relating to a factor of distortion calculation inthe image search apparatus of the second embodiment of the invention.

FIG. 8 is a diagram relating to a data structure managed in the imagesearch apparatus of the second embodiment of the invention.

FIG. 9 is a flow chart (No. 1) relating to a re-searching operation inan image search apparatus of a third embodiment of the invention.

FIG. 10 is a flow chart (No. 2) relating to re-searching operation 1 inthe image search apparatus of the third embodiment of the invention.

DESCRIPTION OF REFERENCE NUMERALS AND SIGNS

-   11 camera-   12 search server-   13 search terminal-   101 multidimensional-feature data producing means-   102 dimension reducing means-   103 approximate-data storing means-   103 a approximate-feature data group-   103 b real-feature data group-   104 search request receiving means-   105 approximate-space searching means-   106 real-space final ranking means

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the invention will be described withreference to the drawings.

Embodiment 1

FIG. 1 is a diagram of an image search apparatus of Embodiment 1 of theinvention. In FIG. 1, 11 denotes a camera which takes an image of aperson, 12 denotes a search server which searches an image including aperson corresponding to designated search conditions, and 13 denotes asearch terminal through which search conditions are designated to causea searching process to be executed. The reference numeral 101 denotesmultidimensional-feature data producing means for extractingmultidimensional feature data for identifying a person, such asface/color/shape, from an image taken by the camera 11, 102 denotesdimension reducing means for reducing the dimensions of themultidimensional feature data extracted by the multidimensional-featuredata producing means 101, thereby producing approximate data, 103denotes approximate-data storing means for making the approximate dataproduced by the dimension reducing means 102 correspondent to themultidimensional feature data before the dimension reduction, and forstoring the data as an approximate-feature data group 103 a and areal-feature data group 103 b, 104 denotes search request receivingmeans for receiving at least an identifier which identifiesmultidimensional feature data of a person to be searched, as a searchkey, 105 denotes approximate-space searching means for, on the basis ofthe search conditions designated by the search terminal 13, calculatingdistances between approximate data corresponding to the search keyreceived by the search request receiving means 104, and pluralapproximate data stored in the approximate-data storing means 103, andfor arranging the distances in order of distance of the calculationresult, i.e., in order of similarity, and 106 denotes real-space finalranking means for again performing a distance calculation with using themultidimensional feature data before the dimension reduction, on a groupof results in which the similarity obtained by the approximate-spacesearching means 105 is high, and for determining final rankings. Thefinal rankings determined by the real-space final ranking means 106 areoutput as a search result.

The person feature data extracted by the multidimensional-feature dataproducing means 101 are image data of a mobile body segmented from animage, or information identifying a mobile body on the basis of a shape,a color, a size, a motion, and the like, or information which identifiesthe shapes and positions of the eyes, nose, and mouth of the face, andthe like. The method of extracting and classifying such featureinformation is well known. For example, the method is strictly describedin “GAZO NO SHORI TO NINSHIKI” (AGUI Takeshi and NAGAO Tomoharu,published by SHOKODO). The person feature data of a face/cloth color,and the like which are produced with using these existing techniques areconfigured by plural elements (referred to as dimensions) foridentifying a person. For example, face feature data are configured byan element group for grasping the whole facial expression, and that forgrasping shapes of specific parts such as eyes/nose/mouth, and the like,in total: several hundred to several thousand dimensions.

FIG. 2 shows the process procedure of the dimension reducing means 102.Hereinafter, its operation will be described.

<Step 201> With respect to input multidimensional feature data (a seriesof [element number, value]), all values are set to absolute values, andsorted in descending order of value. As shown in 2-a, the inputmultidimensional feature data are face feature data having maincomponents in the unit of the whole face/component, data indicating thecolor distribution of the cloth of a person in the form of a color spacehistogram such as RGB/HSV, and data which are obtained by segmenting aregion where an image of a person is taken, and converting the regioninto a frequency. Sets of [element number, value] of elements of theabscissa are given as an input of the multidimensional feature data instep 201. After the sorting, as shown in 2-b, the data are arranged indescending order of absolute value, and elements of the abscissa areproduced as [element number before sorting, value].

<Step 202> The multidimensional feature data are separated (thedimensions are cut) by a designated dimension (R). After the separation,elements within the designated dimension R (those having a largeabsolute value) are output as a series of [element number beforesorting, value] as R1 data, and the portion larger than the designateddimension R is produced as [string of sign bits within typical values V,R2] as R2 data (2-c). As the typical value V of the R2 data, the averagevalue in absolute values of the R2 data, or the maximum value inabsolute values of the R2 data is used. The string of sign bits isproduced as a bit string in which the signs of the elements of R2 areknown in the form that, when the value of an N-th element of R2 ispositive, the bit value=1, and, when the value is negative, the bitvalue=0.

<Step 203> The R1 and R2 data which are produced in step 202 are storedinto the approximate-feature data group 103 a of the approximate-datastoring means 103, and the vector data before sorting which are producedin step 201 are stored into the real-feature data group 103 b, therebyproducing an index. The approximate-feature data group 103 a is a datagroup in which the dimensions are reduced, and hence may be placed on amemory which is accessible at high speed.

FIG. 3 shows the process procedures of the approximate-space searchingmeans 105 and the real-space final ranking means 106. Hereinafter, theiroperation will be described.

<Step 301> Approximate distances between approximate data (3-a)corresponding to the search key received by the search request receivingmeans 104, and plural approximate data (3-b) stored in theapproximate-data storing means 103 are calculated. The pluralapproximate data stored in the approximate-data storing means 103 arearranged in ascending order of approximate distance. In theapproximate-distance calculation, as shown in 3-c, the following processis performed on all the dimensions before sorting:

1) when the dimensions are included in R1 of (3-a) and (3-b), thedistance calculation is performed with using values in R1; and

2) when the dimensions are included in one of R2 of (3-a) and (3-b), theapproximate value of a dimension included in R2 is calculated from thetypical value (V) and sign bit of R2, and the distance calculation isperformed with using the approximate value.

<Step 302> On top M pieces which are arranged in ascending order ofdistance in step 301, real distances with respect to the search keyreceived by the search request receiving means 104 are calculated. Asshown in 3-d, top K pieces in ascending order of real distance areextracted and returned as results. The real distances are calculatedfrom the vector data before sorting which are stored in the real-featuredata group 103 b.

In the process of FIG. 3, a final ranking process is performed in a realspace on the top M pieces which are arranged in ascending order ofapproximate distance. There is a possibility that the omission of searchoccurs at a timing when the data are refined to the top M pieces. FIG. 4shows the process procedure of real-space final ranking means 108 forcausing the omission of search in the process of refining in anapproximate space to be zero. Hereinafter, its operation will bedescribed.

<Step 401> A list of the approximate distances with respect to thesearch key which are produced in the approximate-space searching means105 is obtained. It is assumed that, in the list, the approximatedistances are stored in the ascending order.

<Step 402> Data of a small approximate distance are obtained from theapproximate-distance list. At the timing of obtaining, the correspondingdata are deleted from the approximate-distance list.

<Step 403> Real distances between the data obtained in step 402 and thedata corresponding to the search key are calculated.

<Step 404> The data obtained in step 402 are added to a real-distancelist. In the list, the real distances are stored in the ascending order.

<Step 405> It is determined whether all of the top K pieces of distancesof the real-distance list are smaller than the minimum distance of theapproximate-distance list or not. If Yes, the process is transferred tostep 406, and, if No, the process is transferred to step 402.

<Step 406> The top K pieces of distances of the real-distance list areoutput as a search result of the real-space final ranking means 106.

With reference to FIG. 5, the difference in search result due to theprocess procedures of FIGS. 3 and 4 will be described. In 5-a, examplesof “approximate distance and real distance” between the search key anddata A to H are shown. When a searching process is performed on 5-a inthe flow of FIG. 3, the omission of search occurs in data G as shown in5-b. By contrast, when a searching process is performed in the flow ofFIG. 4, the omission of search does not occur in data G as shown in 5-c.The decision on which of the searching processes of FIGS. 3 and 4 isemployed depends on the relationship between the approximate distanceand the real distance. In the case where “approximate distance≈realdistance” is established in all data, for example, there is nopossibility that the search rankings are largely changed byapproximation, and hence FIG. 4 in which a high-speed process is enabledwhile the omission of search is prevented from occurring is appropriate.By contrast, in the case where “approximate distance<<real distance” isestablished, searching on all data eventually occurs in the process ofFIG. 4 so that the process is slow, and hence the process of FIG. 3 inwhich a high-speed search is enabled while the omission of search issuppressed to some extent is appropriate.

As described above, after search results are refined to some extent inan approximate space where the number of dimensions is suppressed, afinal refinement can be performed in a real space. Even when the numberof dimensions is increased, therefore, an image desired by the user canbe efficiently searched while suppressing the calculation amount. Foreach image, dimensions are reduced while separating into componentswhich highly express a feature of a person, and average components. Evenin the case where an image of a person in which a feature largelyappears in a component that has been assumed not to be high isregistered, therefore, it is possible to search the corresponding personimage flexibly and adequately as compared with the case where dimensionsare reduced with using only components in which the importance ispreviously assumed to be high, such as the eyes/nose/mouth. When thesearching process such as shown in FIG. 4 is introduced, furthermore,the omission of search in the process of refining the search result inan approximate space can be made zero.

As the method of reducing dimensions, the method in which separationinto components highly expressing a feature of a person and averagecomponents is performed by rearranging component values has beendescribed. Alternatively, by using a wavelet transform typified by Haarand the like, the R2 data may be produced from “average components”corresponding to the low frequency, and the R1 data may be produced from“difference components with respect to the average” corresponding to thehigh frequency. In this case, element numbers constituting R1/R2 data donot depend on the input multidimensional feature data, and are fixed.Therefore, it is possible to attain an effect that the calculationamount of the approximate-space searching means 105 can be reduced. Thecase where the differences among the multidimensional feature datastrongly appear in average components can be coped with by acountermeasure that “average components” corresponding to the lowfrequency after the above-mentioned wavelet transform are changed to theR1 data, and “difference components with respect to the average”corresponding to the high frequency are changed to the R2 data.

Embodiment 2

In Embodiment 2, an image search apparatus in which, even when manyomissions of search occur in the process of refining the search resultin an approximate space, the user can easily perform the refiningoperation due to re-searching will be described.

The configuration of Embodiment 2 of the invention is substantiallyidentical with Embodiment 1. Therefore, hereinafter, only the processprocedure of the refining operation due to re-searching which is addedwill be described, and the others will be omitted.

FIG. 6 shows an example in which “factor of distortion due to dimensionreduction” indicating a degree at which a search result obtained by theapproximate-space searching means 105 is changed by the final rankingprocess in the real-space final ranking means 106 is displayed as aresult, and, while referring to “factor of distortion due to dimensionreduction”, the user sets “number of dimensions to be used” and “piecenumber of results to be refined” as re-search conditions, in theapproximate-space searching means 105. In the figure, 6-a is an initialsearch condition designating screen in which a search key of a person tobe searched and a search range of time/place are initially designated,6-b is a search result displaying screen on which a search result isdisplayed together with “factor of distortion due to dimensionreduction”, and 6-c is a re-search condition designating screen inwhich, as the next search conditions, the user designates one of threere-search methods [1) adjustment of the used dimension number (number ofdimensions to be used), 2) adjustment of the approximate range (piecenumber of results to be refined), and 3) the above-mentioned 1) and 2)are not performed, and next K pieces are output]. The operations of 6-band 6-c are repeatedly performed.

Referring to FIG. 7, an example of a calculation of “factor ofdistortion due to dimension reduction” will be described. In the figure,7-a shows an example of the omission of search (data L, G) which occursin the case where a search is performed in the configuration such asshown in FIGS. 1, and 7-b shows a difference between search resultswhich are obtained in the approximate-space searching means 105 and thereal-space final ranking means 106 when the omission of search occurs.In 7-b, when the value of M in the top M pieces which is a threshold ofrefinement in an approximate space is made large, the possibility of theomission of search is reduced. However, the degree at which the value ofM is to be increased by the user cannot be determined. Consequently,when a) a ratio at which the top K pieces obtained in an approximatespace are included in the top K pieces in the final rankings, or b) aranking ratio of “total sum of final rankings (i.e., K*(K+1)/2)/totalsum of rankings in an approximate space” to the top K pieces of data inthe final rankings is used as “factor of distortion due to dimensionreduction”, it is possible to inform the user of the degree at whichdistortion due to dimension reduction occurs. As the value of a) or b)is smaller, the degree of distortion is larger.

Next, the operation in which the re-search conditions are designated bythe user with reference to “factor of distortion due to dimensionreduction” will be described in detail. As shown in 6-c, in designationof the re-search conditions, there are three patterns of 1) adjustmentof the used dimension number, 2) adjustment of the approximate range,and 3) the next K pieces are output.

The used dimension number of 1) means the number of elements of the R1data used in the approximate-space searching means 105. When the numberof elements of the R1 data is increased, “factor of distortion due todimension reduction” can be reduced. In order that, when the useddimension number is adjusted, the approximate-space searching means 105is enabled to change the element number of the R1 data and perform are-searching process, a data structure corresponding to plural cutdimensions (R_a, R_b, R_c) such as shown in FIG. 8 is previouslyproduced. The data structure is not required to be produced for each ofthe plural cut dimensions. As shown in 8-b, it can be coped with bypreparing the data structure so as to correspond to the cut dimension(R_c) having a large element number of the R1 data.

The adjustment of the approximate range of 2) is performed in order toadjust the range to be refined in an approximate space (the value of Min the top M pieces). Even when “factor of distortion due to dimensionreduction” is large, the omission of search can be prevented fromoccurring by increasing the value of M.

The next K pieces of 3) are used when the user determines that “factorof distortion due to dimension reduction is small” or “factor ofdistortion due to dimension reduction is large but the adjustment 1) or2) is difficult”.

As described above, the user refers to “factor of distortion due todimension reduction”, and adjusts “number of dimensions to be used” and“piece number of results to be refined” by means the approximate-spacesearching means, whereby a re-searching operation in which the omissionof search in a process of refining the search result in an approximatespace is suppressed can be realized. Alternatively, a configurationwhere re-search condition designating means refers to “factor ofdistortion due to dimension reduction” output by the real-space finalranking means to automatically re-designate the search conditions may beemployed. According to the configuration, an image can be searched moreefficiently.

Embodiment 3

In Embodiment 3, an image search apparatus which is different fromEmbodiment 2, and in which, even when many omissions of search occur inthe process of refining the search result in an approximate space, theuser can easily perform the refining operation due to re-searching willbe described.

The configuration of Embodiment 3 of the invention is substantiallyidentical with Embodiment 1. Therefore, hereinafter, only the processprocedure of the refining operation due to re-searching which is addedwill be described, and the others will be omitted.

FIG. 9 shows an example in which, when the user designates correctnessor incorrectness of the search result, the approximate-space searchingmeans 105 increases weightings of element numbers used in approximatedata designated as correct, and decreases weightings of element numbersused in approximate data designated as incorrect, and then the distancecalculation in an approximate space is again performed. As shown in 9-b,element numbers which highly express a feature of a person are differentfor each of search results. From results which are designated ascorrect/incorrect by the user, therefore, element numbers from whichcorrectness is to be derived, and those from which incorrectness is tobe removed are extracted and weighted, whereby a re-searching process inwhich the omission of search is suppressed can be realized. Theweighting means an operation of, in the distance calculation of eachdimension in 3-c of FIG. 3, providing a result of the distancecalculation with a weighting factor.

FIG. 10 shows an example in which, when the user designates a searchresult as incorrect, “number of dimensions to be used” and “piece numberof results to be refined” are automatically adjusted in theapproximate-space searching means 105 so that the data designated asincorrect are not output in the real-space final ranking means 106. In10-b, when the user designates “data H/E=incorrect”, the process isperformed while the value of M in the top M pieces which is a thresholdof refinement in an approximate space is made large, or the number ofdimensions to be used is increased, until an element which is smaller indistance than the data H/E.

As described above, the user designates correctness/incorrectness foreach image, and approximate-distance calculation parameters (the weightof the element number, the number of dimensions to be used, and theapproximate range) in the re-searching process are automaticallyadjusted, whereby a re-searching operation in which the omission ofsearch in a process of refining the search result in an approximatespace is suppressed can be realized.

As described above, the image search apparatuses and image searchmethods of the embodiments of the invention have an effect that, aftersearch results are refined to some extent in an approximate space wherethe number of dimensions is suppressed, a final refinement is performedin a real space, whereby, even when the number of dimensions isincreased, an image desired by the user can be searched efficiently andadequately. The apparatus and method can be applied also to uses ofview, search, and edition on contents (still image or motion picture)obtained by a personally in travel, athletic meet, or the like, inaddition to a monitoring use in which all behaviors of as a shoplifter,a lost child, or a person who has lost an article are grasped with usingplural cameras.

While the invention has been described in detail with reference tospecific embodiments, it will be obvious to those skilled in the artthat various changes or modifications may be made without departing fromthe sprit and scope of the invention.

The present application is based on Japanese Patent Application (No.2005-370613) filed Dec. 22, 2005, and its disclosure is incorporatedherein by reference.

INDUSTRIAL APPLICABILITY

The invention has an effect that, after search results are refined tosome extent in an approximate space where the number of dimensions issuppressed, a final refinement is performed in a real space, andtherefore, even when the number of dimensions is increased, an imagedesired by the user can be searched efficiently. The invention is usefulin an image search apparatus, image search method, or the like which cansort out a desired image at high speed from an image group of largecapacity with using multidimensional feature data (a face, a color, andthe like) extracted from an image.

1. An image search apparatus comprising: a dimension reducing unit thatreduces dimensions of multidimensional feature data extracted from animage, to produce approximate data; an approximate-data storing unitincluding a memory that stores the approximate data produced by thedimension reducing unit corresponding to the multidimensional featuredata used to produce the approximate data; a search request receivingunit that receives at least an identifier which identifiesmultidimensional feature data of a person to be searched, as a searchkey; an approximate-space searching unit that calculates distances fromapproximate data corresponding to the search key received by the searchrequest receiving unit, to plural approximate data stored in theapproximate-data storing unit, based on search conditions, and arrangesthe distances in order of similarity; and a real-space final rankingunit that performs a further distance calculation of distances from themultidimensional feature data used to produce the approximate dataproduced by the dimension reducing unit, to corresponding data of thesearch key, on a group of results in which the similarity obtained bythe approximate-space searching unit is high, determines final rankings,and outputs the final rankings as a search result, wherein thereal-space final ranking unit performs a final ranking process on top Mpieces (M: natural number) in which the similarity obtained by theapproximate-space searching unit is high, and outputs top K pieces (K<M)thereof as the search result.
 2. The image search apparatus according toclaim. 1, wherein the dimension reducing unit rearranges elementsconstituting the multidimensional feature data in descending order of anabsolute value, and produces “element numbers, values” of a top N pieces(N: natural number) thereof as the approximate data.
 3. The image searchapparatus according to claim 1, wherein, from a result of a wavelettransform of input multidimensional feature data, the dimension reducingunit produces values of top N pieces of high-frequency components orlow-frequency components, as the approximate data.
 4. The image searchapparatus according to claim 2, wherein multidimensional feature dataother than the top N pieces of data obtained by the dimension reducingunit are produced as (typical values, sign bits), and managed as theapproximate data.
 5. The image search apparatus according to claim 3,wherein multidimensional feature data other than the top N pieces ofdata obtained by the dimension reducing unit are produced as (typicalvalues, sign bits), and managed as the approximate data.
 6. The imagesearch apparatus according to claim 1, wherein the real-space finalranking unit performs a distance recalculation using themultidimensional feature data used to produce the approximate dataproduced by the dimension reducing unit, in descending order of elementsin which the similarity obtained by the approximate-space searching unitis high (i.e., in ascending order of the approximate distance),completes the process at a timing when top K pieces of actual distanceswhich are obtained by the recalculation and in which the similarity ishigh are smaller than an approximate distance of all data that are notsubjected to the distance recalculation, and outputs the top K pieces ofdistances as the search result.
 7. The image search apparatus accordingto claim 1, wherein the real-space final ranking unit outputs “factor ofdistortion due to dimension reduction” indicating a degree at which aresult obtained by the approximate-space searching unit is changed bythe final ranking process, as a result.
 8. The image search apparatusaccording to claim 7 further comprising a re-search conditiondesignating unit that designates “number of dimensions to be used” and“piece number of results to be refined” as search conditions to be usedby the approximate-space searching unit.
 9. The image search apparatusaccording to claim 1 further comprising a correctness/incorrectnessdesignating unit that designates correctness or incorrectness of thesearch result output by the real-space final ranking unit.
 10. The imagesearch apparatus according to claim. 9, wherein the approximate-spacesearching unit sets, as “correct element number group”, element numbersof approximate data which are designated as correct by thecorrectness/incorrectness designating unit, sets, as “incorrect elementnumber group”, element numbers of approximate data which are designatedas incorrect, and, in the distance calculating process using approximatedata, increases weightings of element numbers included in “correctelement number group”, and decreases weightings of element numbersincluded in “incorrect element number group”.
 11. The image searchapparatus according to claim 9, wherein the approximate-space searchingunit automatically adjusts “number of dimensions to be used” and “piecenumber of results to be refined” which are the search conditions so thatthe approximate data designated as incorrect by thecorrectness/incorrectness designating unit are not output in thereal-space final ranking unit.
 12. An image search method comprising:approximate-space searching step of using approximate data produced byreducing dimensions of multidimensional feature data extracted from animage, comparing the approximate data produced by reducing dimensions ofthe multidimensional feature data to approximate data corresponding to asearch key, and, based on a result of said comparing, arranging imagegroups stored in an image storing unit in order of similarity with animage to be searched; and real-space final ranking step of performing adistance re-calculation using the multidimensional feature data used toproduce the approximate data produced by said reducing dimensions, on agroup of results in which the similarity obtained in theapproximate-space searching step is high, and determining finalrankings, wherein in the real-space final ranking step, a final rankingprocess is performed on top M pieces (M: natural number) in which thesimilarity obtained by the approximate-spaced searching step is high,and top K pieces (K<M) are output as the search result.
 13. The imagesearch method according to claim 12, wherein elements constituting themultidimensional feature data are rearranged in descending order of anabsolute value, and “element numbers, values” of top N pieces (N:natural number) thereof are produced as the approximate data.
 14. Theimage search method according to claim 12, wherein, from a result of awavelet transform of input multidimensional feature data, values of topN pieces of high-frequency components or low-frequency components areproduced as the approximate data.
 15. The image search method accordingto claim 13, wherein multidimensional feature data other than the top Npieces of data obtained by the dimension reducing unit are produced as(typical values, sign bits), and managed as the approximate data. 16.The image search method according to claim 14, wherein multidimensionalfeature data other than the top N pieces of data obtained by thedimension reducing unit are produced as (typical values, sign bits), andmanaged as the approximate data.
 17. The image search method accordingto claim 12, wherein, in the real-space final ranking step, a distancerecalculation is performed using the multidimensional feature data usedto produce the approximate data produced by said reducing dimensions, indescending order of elements in which the similarity obtained by theapproximate-space searching step is high (i.e., in ascending order ofthe approximate distance), the process is completed at a timing when topK pieces of actual distances which are obtained by the recalculation andin which the similarity is high are smaller than an approximate distanceof all data that are not subjected to the distance recalculation, andthe top K pieces of distances are output as the search result.
 18. Theimage search method according to claim 12, wherein, in the real-spacefinal ranking step, “factor of distortion due to dimension reduction”indicating a degree at which a result obtained by the approximate-spacesearching step is changed by the final ranking process is output as aresult.
 19. The image search method according to claim 18 furthercomprising a re-search condition designating step of designating “numberof dimensions to be used” and “piece number of results to be refined” assearch conditions to be used by the approximate-space searching step.20. The image search method according to claim 12 further comprising acorrectness/incorrectness designating step of designating correctness orincorrectness of the search result output by the real-space finalranking step.
 21. The image search method according to claim 20,wherein, in the approximate-space searching step, element numbers ofapproximate data which are designated as correct by thecorrectness/incorrectness designating step are set as “correct elementnumber group”, element numbers of approximate data which are designatedas incorrect are set as “incorrect element number group”, and, in adistance calculating process using approximate data, weightings ofelement numbers included in “correct element number group” areincreased, and weightings of element numbers included in “incorrectelement number group” are decreased.
 22. The image search methodaccording to claim 20, wherein, in the approximate-space searching step,“number of dimension to be used” and “piece number of results to berefined” which are search conditions are automatically adjusted so thatthe approximate data designated as incorrect by thecorrectness/incorrectness designating step are not output in thereal-space final ranking step.